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Foreword 


This paper summarizes findings from a consensus panel on using data from learning management 
systems and software to help educators understand and improve student engagement during 
remote instruction. 


Student engagement is central to learning. As a research synthesis by the University of Chicago 
Consortium on School Research notes, “Curriculum, standards, and tests don’t matter if 
students are not participating in class and investing themselves in the learning opportunities 
that teachers prepare.”! 


Research shows that students are more engaged and successful when they experience supportive 
classroom and school environments; yet even before COVID-19, many students, particularly 
students of color and students living in poverty, did not have access to learning conditions 
that would promote high levels of engagement.? Remote learning has only exacerbated those 
inequities, as evidenced by high absenteeism rates, increased failure rates, and the perceptions 
of students themselves. 


A Youth Truth survey of more than 20,000 students in grades 5 through 12 found that only 41 
percent were able to motivate themselves to do schoolwork while their school buildings have 
been closed.* Black and Hispanic students reported facing more obstacles to remote learning 
than white and Asian students.° 


Teachers can tap into a number of signals to know if they are effectively engaging students 
during in-person instruction. These range from students’ postures and facial expressions to 
their participation in class discussions and their focus on classroom activities. Unfortunately, 
many of these signals are absent during remote learning. 


1 Elaine M. Allensworth et al., Supporting Social, Emotional, & Academic Development: Research Implications for 
Educators (Chicago, IL: University of Chicago Consortium on School Research, 2018). 

2 Ibid. 

3 Laura Meckler and Hannah Natanson, “‘A lost generation’: Surge of research suggests students sliding backward, 
most vulnerable worst affected,” The Washington Post, December 6, 2020; Alec MacGillis, “The Students Left 
Behind by Remote Learning: Has a desire to keep coronavirus out of schools put children’s long-term well-being at 
stake?” ProPublica, September 28, 2020. 

4 Students Weigh In: Learning and Well-Being During COVID-19 (San Francisco, CA: YouthTruth, 2020). 

5 Ibid. 
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That’s why the Center on Reinventing Public Education brought together a group of researchers, 
practitioners, and technology and curriculum providers to explore how learning management 
systems and courseware, as well as student information systems, could help provide teachers 
and school systems with data to better measure student engagement and learning both in the 
short term and in the future, with the goal of better supporting students. 


The consensus panel was asked to address three questions: 


1. What data are important to measure student engagement and learning and how much 
of that data are currently available through existing learning management systems and 
software? 


2. What is needed to make that data more accessible and actionable for teachers and 
school administrators? 


3. What features could be built into learning management systems and software in the 
future to better understand and assess student engagement and learning? 


The panel concluded: 


¢ Learning software and platforms that many districts use already provide a host of data on 
students’ access to technology, participation in learning, and even aspects of the learning 
environment. But stand-alone data points are hard to make sense of; most districts lack 
the data systems and technical capacity to integrate data across multiple sources and 
software platforms to place specific data points in context. Data on students’ assignment 
completion rate is more meaningful and actionable when combined with information on 
students’ prior performance, technology and internet access, and household conditions. 


* To make data accessible and actionable, districts need more consistent routines around 
data retrieval, analysis, and reporting. Platforms can help districts by surfacing the right 
data at the right time and presenting it in simple, teacher-friendly dashboards. 


¢ Technology and education leaders alike have creative ideas about new features and data 
that would further advance our understanding of students’ engagement in learning. 
Technology firms are eager to build more functional tools for users but they need both 
guidance and coordination from the field to build solutions that work for more than a 
single use in a single school system. Tech providers should codesign such solutions with 
the ultimate end users: educators and school system leaders. 


Student engagement was a problem before the pandemic. This is an opportunity to design 
solutions that will help leaders and educators long after the crisis passes. 
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Software Platforms Used in Schools 


Learning Management System: A web-based platform that enables educators to create 
and deliver online content to students, monitor what students are doing and how they are 
progressing through customized assessments and assignment completion, and use that 
information to grade student performance. Teachers also also create discussion groups to 
promote student communication, engagement, and feedback. Views can be customized 
for different users, such as teachers, students, and parents. 


Learning Courseware: Packaged online classes, curricula, or courses bundled together 
with the various lessons, tests, and other materials needed for students to learn digitally 
and track progress. 


Student Information System: A web-based platform that helps schools and school systems 
manage student data that would otherwise be spread across multiple locations for ease 
of use. This might include registering students; documenting grades, transcripts, and test 
results; building student schedules; and tracking attendance. Views can be customized for 
different users, such as district and school administrators and teachers. 


Some data on student engagement are already available, but 
often hard to make sense of on its own 


The panel identified the following data needs: 


¢ Information about students’ access to learning, such as technology devices and stable 
internet connections. 


« Real-time access to disaggregated data on student participation in remote learning. 


¢ Data about how students experience the learning environment, so that teachers can 
better support students. 


¢ Reliable data on student learning that can inform teachers’ instruction. 


Access: Do students have access to the devices and connectivity they 
need to engage? 


School systems should not overlook the importance of basic data: Do students even have 
the tools to participate in remote learning? Information about whether students engage in 
remote learning must sit alongside data about whether students have access to the technology 
devices and stable internet necessary to participate. Such data should be broken out by student 
subgroups and, ideally, include information about other barriers to participation, such as housing 
or food insecurity. 


The Wisconsin Department of Public Instruction, for example, has developed a Digital Equity 
Dashboard with voluntary responses from families regarding their access to digital devices 
and the internet in their homes. States participating in the Ed-Fi Alliance, such as Wisconsin, 
have included new fields for such data in their student information systems to make it readily 
accessible to districts and schools. 
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In its Data Portal for schools, New Visions for Public Schools in New York City puts information 
about students’ computer, internet, and smartphone access alongside information about student 
engagement and assignment completion to help teachers and school leaders understand the 
barriers to participation and how they might be addressed. To build these features into its Data 
Portal, the nonprofit had to pull information from multiple data systems within the New York 
City public schools. 


This kind of integration isn’t always easy—districts tend to work with multiple software platforms 
simultaneously with information locked in silos. It requires consistent ways to link information 
about students and schools across platforms and better application platform interfaces (APIs) 
that allow for integration across multiple platforms. 


Fortunately, some platforms are poised to provide this integration. Districts using Infinite 
Campus, for example, can pull information about assignment completion from its learning 
management system, alongside data about student demographics, attendance, whether 
the student has moved in the middle of the year, or whether the household membership or 
guardianship has changed from its student information system. 


Consistent definitions of key data elements at the data dictionary level across systems helps. 
Infinite Campus, for example, requires all its customers to map localized attendance codes to a 
centralized Infinite Campus standard. 


A number of districts also have data-sharing agreements with community-based organizations, 
such as social service agencies, that could help identify which students have received services in 
prior years and who may need additional support. Such information could help shed light on why 
some students are not showing up online and how to provide comprehensive services for those 
students. Yet teachers typically do not have access to such data to provide a comprehensive 
view of students’ needs. In Kentucky, Infinite Campus has an entire social services section of the 
learning management system to help school social workers track which students are receiving 
which services. 


Getting data about the broader context of students’ lives requires integrating technical systems 
in order to merge data, as well as partnerships among organizations to arrive at data-sharing 
agreements. Both pose coordination challenges. 


Participation: Are students logging on? Are they completing 
assignments? 


One measure of engagement is who is logging onto remote learning platforms, either during 
synchronous or asynchronous instruction. Last spring, many school systems failed to collect 
this data though it’s readily available. To make it more useful, educators want to know who 
is logging on, for what classes or activities, and for how long, which would provide a better 
indicator of participation than having teachers take attendance each morning. 


Educators cautioned, however, that information about which students are logging on is not a 
sufficient measure of engagement. One urban school administrator noted, “A lot of our teachers 
have found that students are logging on, but they’re just boxes with names and not participating. 
Particularly in our secondary schools ...a lot of students are taking the opportunity to get jobs 
and, in fact, logging on through their phones while working.” The district is working with a small 
group of teachers to better define what “presence” in a virtual classroom actually means. 
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Another signal of participation is whether students are completing the lessons assigned to them, 
either by a particular learning platform or by a teacher. New Visions, which supports a network 
of public district and charter schools in New York City, pulls analytics from Google Classrooms 
for schools in its network. These include students’ engagement in Google Classrooms in the last 
five days, the number of assignments a teacher has given versus the number of assignments a 
student has completed, and whether a student has logged onto Google Hangouts over the last 
five days to interact with teachers or peers. 


Teachers or school leaders can identify which information they’d like to review, group students 
(such as 9th graders who are housing insecure and have engaged for three days or less in the 
last week or Black students who have submitted 10 or fewer assignments), and then assign 
students to a support, such as a study hall, office hours, or a small group to address assignment 
completion. Individual schools have used the data to decide how many and what types of 
assignments to give students on a weekly basis, while teachers have used the data to help 
students set goals or put in place a plan to address assignment completion. 


Some digital curricula also can provide teachers and administrators with insights into student 
participation. Zearn Math’s classroom Pace Report offers teachers a real-time view of how 
many Zearn Digital Lessons each student has completed, their time to complete each lesson, 
and their aggregate weekly time spent to help identify groups of students who need more time 
to meet their weekly goal. 


Learning Environment: What are the conditions of remote classrooms 
that promote student engagement? 


Hearing directly from students about how they experience remote instruction is vital. “Student 
voices are crucial to get anything right at this point,” said one participant, “to hear what they’re 
experiencing and what they’re feeling.” 


Research has found that students are more motivated to learn and more capable of learning 
when they experience certain learning conditions, such as a caring teacher, work they view as 
relevant, and a sense of community and respect from their peers.® Unfortunately, teachers rarely 
get insights into how their students are experiencing the learning environment. The Copilot- 
Elevate student survey, developed by PERTS and the University of Chicago Consortium on 
School Research, enables teachers to measure the quality of key learning conditions—teacher 
caring, student voice, meaningful work, feedback for growth, classroom belonging, and affirming 
cultural identity—on a regular basis and provides teachers with practical recommendations for 
cultivating each condition.’ Students respond to such statements as, “This week, my teacher 
challenged me to learn as much as | can,” and “This week, my teacher treated me with respect.” 
When equipped with this student feedback and actionable recommendations, the vast majority 
of teachers successfully improved these learning conditions.® 


In February, Infinite Campus will release an update that enables teachers to ask students 
questions about their social-emotional learning to help guide instruction, such as how they 
are feeling that day, and whether they think they’ll do well on an assignment. The questions, 
developed with the American Institutes for Research, can be integrated into a particular unit, 


6 Allensworth et al., Supporting Social, Emotional, & Academic Development: Research Implications for Educators. 
7 David Paunesku and Camille A. Farrington, “Measure Learning Environments, Not Just Students, to Support 
Learning and Development,” Teachers College Record 122, no. 14 (2020). 

8 Ibid. See also: Sarah Gripshover and David Paunesku, How can schools support academic success while fostering 
healthy social and emotional development? (Stanford, CA: PERTS, Stanford University, 2019). 
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lesson, or assignment or into the daily work flow, such as during morning meetings. New Visions 
also is working to incorporate social-emotional survey data alongside other academic data to 
help schools think about how best to address student needs. 


During the pandemic, many schools and school systems have created regular check-ins with 
students, such as morning meetings or mentoring or advisory systems, to better support 
students. Minneapolis Public Schools, for example, requires every teacher to host a morning 
meeting to check in with students on how they’re doing and how they’re feeling about their 
learning. But that information is not captured by learning management systems in a systematic 
Way. 


Last spring, the San Antonio Independent School District developed a student interaction 
tracker to collect information on the multiple ways students may participate in online learning, 
whether through classes, phone calls from the teacher, or using Classlink to access digital 
resources. The app enabled teachers and support staff to log every interaction with a student, 
as well as whether students submitted assignments and participated in classes. An interactive 
dashboard updated student interactions daily, with the view dependent on the user’s job. A 
teacher could track the engagement and progress of students in her classroom, for example, to 
see if they needed additional support or specific help, while a district administrator would have 
amore high-level summary to examine trends. The information helped inform school reopening 
plans for this school year. 


Learning: Is there evidence that students are actually learning and 
making progress? 


Ultimately, teachers and school systems want to know whether students are learning and making 
progress against rigorous expectations, not just whether they are engaged in their instruction. 
Yet many districts are struggling with how best to collect evidence of student learning during 
remote instruction. 


Teachers and school administrators using i-Ready, a comprehensive assessment and instruction 
program from Curriculum Associates, can access data on how much time students are spending 
on each lesson, their pass rates, their proficiency levels in reading and math, and where they are 
struggling—but only if districts select that particular tool. NWEA’s MAP assessments, another 
widely used suite of online assessments, enables teachers and school administrators to track 
student growth over the course of a year and to pinpoint specific skills gaps. To help teachers 
personalize instruction, the MAP Accelerator for grades 3 through 8 automatically integrates 
class rosters with students’ growth scores and Common Core-aligned Khan Academy content 
to generate lessons, instructional videos, and practical problems for students based on their 
needs. 


Zearn Math’s Tower Alerts Report for teachers highlights details about exactly what students in 
their classroom are struggling with during digital lessons. Zearn Admin Reports offer coaches 
and other administrators the opportunity to monitor schoolwide progress through grade-level 
content and student productivity during independent work. 


Policymakers and decisionmakers also can use such data to understand and address the 
differential impact of remote learning on different student subgroups. A Zearn analysis of 
students’ progress in online math coursework from January 2020 to January 2021 found that 
students in high-income schools increased their progress by 14.9 percent, while students in 
low-income schools had a 25.6 percent decrease. 
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Platforms like Nearpod enable teachers to upload videos, slide presentations, and other 
materials onto an interactive platform and gain insights into student learning through polls, 
games-based quizzes, and checks-for-understanding. But it’s not clear how schools or school 
systems can use that daily, formative data to identify broader patterns regarding students’ 
strengths or learning gaps. 


Some panelists wondered whether teachers’ grades or exit tickets at the end of remote classes 
could be more routinely entered into learning management systems. For example, districts 
could require teachers to enter students’ grades on assignments on a regular basis. While 
there’s nothing especially exciting or innovative about such data, noted one panelist, they 
can provide important early warning signs about whether students need additional support 
or intervention. “Teachers already have to keep a gradebook,” he noted. “Doing so through 
the learning management system would be much more useful because of the transparency it 
would introduce. | think it would be the single most meaningful and universal signal of whether 
or not students are learning.” 


Routines for retrieval, analysis, and reporting would help make 
data more actionable 


A bigger need than adding new data to learning management systems is helping educators 
make use of the data that’s already there. That requires building stronger routines around data 
retrieval and analysis that make better use of the most obvious information (such as participation, 
log-in information, and assignment completion) and standardizing some of those data reports. 


Learning management systems can help by surfacing the right data at the right time based on 
a focused set of information. Infinite Campus’s early warning system for dropout prevention, 
for example, tracks about 75 indicators but organizes them around attendance, behavior, 
academics, and home/enrollment stability to generate machine-learned warning flags for 
individual students. Infinite Campus is interested in developing such automated early warning 
systems for other educational outcomes. 


Few school systems, except for the larger and wealthier districts, have data engineers who can 
get into the raw data and write a query to do this type of analysis. One solution would be to 
convene vendors, researchers, and districts to identify the highest leverage and most desired 
use cases, so that vendors would have an incentive to develop products that could be widely 
used, beyond a single school system. (IES has recently proposed such an effort.) 


In particular, educators need simpler data dashboards, as well as protocols to help them 
see patterns in the data and know what steps to take in response to those patterns. A good 
platform, said one participant, brings together multiple sources of data, adds a layer of analysis, 
and gives educators the ability to break down the data by many different subgroups to surface 
more granular themes. 


“We have a lot of data on individual students and probably more now than ever,” said one 
participant. “But what do we do with that to put it in the right context? How do we filter through 
what’s happening into something that’s actionable and .. . useful to keep kids progressing in 
their learning goals?” 


“The data on who is chatting during class, who is posting in Google Classroom, etc., | think are 
quite telling in terms of engagement,” said another participant, “but | think a teacher would 
benefit from help summarizing that real-time data in a way that they can actually use.” 
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The BARR Center, for example, has developed protocols for teacher teams in its network to 
meet weekly to discuss student-level data in order to identify students who need interventions 
or acceleration. 


Minnesota’s Early Indicator and Response System walks school-based teams through a seven- 
step process designed to keep students in school, starting with ensuring there is a tool and a 
process for collecting and updating data regularly on each student. 


Similarly, New Visions found that just giving educators access to data does not mean they’re 
going to use it. The nonprofit has saved educators time by automating the analysis of data that 
teachers and school counselors once had to calculate themselves—such as which courses or 
Regents exams a student still needed to pass to graduate. It then created routine processes, 
called strategic data check-ins, focused on high-stakes tasks that schools must complete, such 
as scheduling students for high school courses or Regents exams and study sessions. And 
it developed protocols to help school leaders and teacher teams at the school site uncover 
patterns in the data to design interventions for individuals or groups of students. 


“Throwing more data at teachers isn’t going to solve the problem unless there’s a really 
strong, actional plan for educators to use the data meaningfully for improvement,” noted 
one participant. “The behavior and human-change aspect of this cannot be overlooked,” said 
another participant. “It’s probably the biggest obstacle in doing this work. The messaging and 
the support can’t be overlooked.” 


Given the multiple users of data generated from learning management and student information 
systems, the type of data that might be most meaningful, how it’s presented, and how often 
should vary based on the end user, such as teachers, schools, districts, policymakers, and 
researchers. 


Groups like the Achievement Network, or ANet, have recommended a tiered approach to 
collecting data that prioritizes data closest to instruction. Some data, such as attendance and 
participation in online classes, would be collected for all students; other data, such as student 
interviews to understand why a student is not participating or completing assignments, would 
be limited to those most at risk. While a school might conduct weekly data reviews with a 
focus on a few students with the greatest needs, it might conduct monthly data reviews with 
a particular focus on some students who are not fully participating in remote learning, and 
quarterly data reviews on the progress of all students. Without a commitment from the central 
office, participants noted, it’s difficult to build or sustain a culture of data use at the teacher or 
school level. 


Creative new features are in view but tech providers need to 
codesign them with system leaders 


We identified a few concrete changes that could be made to integrate additional data about 
student engagement into learning management systems and software, particularly to address 
unconscious bias. 


For example, could learning management platforms capture teacher actions or responses— 
such as whether teachers call on students differently by race or gender—so that teachers or 
school administrators could review such actions for unconscious bias? An open-source video 
conferencing solution that includes an API for who’s talking, for example, could help teachers 
reflect upon their interactions with students. 
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Vendors also could build specific recommendations to change practice into learning 
management systems. Right now, such platforms may be able to identify students who 
aren’t engaged, and possibly the circumstances in which students tune out, but generally do 
not provide guidance on how to change teacher practice. One idea would be to build more 
evidence-based practice recommendations into such systems. Platforms also could play a role 
in connecting districts with high levels of student engagement with districts that are struggling. 


Tech providers should be codesigning such solutions with the end users. “I think it has to be 
done in collaboration with people on the ground,” said one participant. “You need to understand 
their systems and it should complement what they are already doing. It shouldn’t feel like an 
add-on but rather a support.” 


Conclusion 


Educators need to understand the value and limitations of learning management systems in 
boosting student engagement. Such systems can help identify the broad contours of student 
engagement and direct teachers, principals, and system leaders to potential soft spots, but 
the data aren’t actionable on their own. They must be viewed in context with other data about 
students and their learning environment. 


Even the data that do exist are often underused, in large part because of limited district capacity 
to surface and integrate the data and a lack of useful data-integration tools. As technology 
tools find a more rooted position in school systems, exciting potential exists not only to build 
more integrated systems with data dashboards and accompanying routines but also to add 
new features that draw more from students about their experiences and the nature of their 
engagement. 


Student engagement was a problem even prior to COVID-19. We should use this opportunity 
to rethink instruction in ways that genuinely engage many more students, rather than trying to 
motivate them to engage in instruction as currently delivered. That may require a different kind 
of conversation with students, parents, and educators—and an entirely different type of data 
collection—to envision what’s possible. 
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